SVM Further Exploration

After the previous analysis, we have developed an interest in SVM. We will now explore the SVM algorithm in more depth and apply it to a different dataset.

Support Vector Machine (SVM) is a powerful machine learning algorithm used for binary classification tasks. It aims to find the optimal hyperplane that best separates data points of different classes in a high-dimensional feature space. The key idea of SVM is to identify the 'support vectors', which are the data points closest to the decision boundary or hyperplane. These support vectors play a critical role in determining the position and orientation of the hyperplane, making SVM a robust and effective method for solving binary classification problems.

Cross Validation

Cross-validation is a technique used in machine learning to evaluate the performance of a model. It involves dividing the dataset into multiple folds, training the model on a subset of the folds, and evaluating it on the remaining fold. This process is repeated multiple times to obtain a reliable estimate of the model's performance. Cross-validation is commonly used for model evaluation, selection, and performance estimation.

In this further exploration, we utilize cross-validation for analysis.

Importing Libraries and Data

import pandas as pd
import numpy as np
import matplotlib.pyplot as plt
import seaborn as sns
from sklearn.svm import SVC

## cross validation
from sklearn.model_selection import cross_val_score

X = pd.read_csv('data/cleaned/X.csv')
y = pd.read_csv('data/cleaned/y.csv')

Exploring the Data with 2 Features

To gain a preliminary understanding of SVM, we choose to analyze two parameters, which allows for better visualization and intuitive comprehension.

X_important = X[["meanfun", "IQR"]]

Use 2 Features

This function takes in a support vector machine (svm) classifier, a feature matrix X, a target vector y, and a plot title. It then generates a plot of the decision boundary for the SVM based on the "meanfun" and "IQR" features in the X matrix, as well as a scatter plot of the data points colored by their target value.

First, it sets the plot size to be 20x20 units. It then generates a 100x100 grid of values for "meanfun" and "IQR" using np.linspace, and creates a matrix of all possible combinations of these values using np.meshgrid. It then applies the SVM decision function to each point on this grid, which generates a matrix of predicted target values. The plot function then draws the decision boundary at values of -1, 0, and 1, and colors the regions between them with different line styles.

Finally, it generates a scatter plot of the data points in X, using only the "meanfun" and "IQR" columns, and colors each point based on its corresponding target value in y. It then sets the x- and y-axis labels, and the plot title, before displaying the plot using plt.show().

def draw_svm_plot(svm, X, y, title):
    plt.figure(figsize=(20, 20))
    x_plot = np.linspace(X["meanfun"].min(), X["meanfun"].max(), 100)
    y_plot = np.linspace(X["IQR"].min(), X["IQR"].max(), 100)
    X_plot, Y_plot = np.meshgrid(x_plot, y_plot)
    xy = np.vstack([X_plot.ravel(), Y_plot.ravel()]).T
    Z = svm.decision_function(xy).reshape(X_plot.shape)
    plt.contour(X_plot, Y_plot, Z, colors="k", levels=[-1, 0, 1], alpha=0.5, linestyles=["--", "-", "--"], linewidths=3)
    sns.scatterplot(data=X_important, x="meanfun", y="IQR", hue=y.squeeze())
    plt.xlabel("meanfun")
    plt.ylabel("IQR")
    plt.title(title)
    plt.show()

Linear Kernel

The linear kernel is the simplest kernel. The formula for the linear kernel is: $$ K(x, x') = x^T x' $$

In this code, a support vector machine (SVM) model with a linear kernel is trained using the feature matrix X_important and target vector y. The code then performs 5-fold cross-validation to evaluate the model's accuracy. The average accuracy across all folds is then printed to the console.

Finally, the function draw_svm_plot() is called to plot the decision boundary of the trained SVM model along with the data points in X_important, colored by their corresponding target values in y. The plot is titled "Linear Kernel".

from sklearn.svm import SVC

svm_linear = SVC(random_state=87, kernel="linear").fit(X_important, y.squeeze()) # Create SVM model object
scores = cross_val_score(svm_linear, X_important, y.squeeze(), cv=5)  # Perform 5-fold cross-validation
svm_linear_accuracy = scores.mean()  # Get the average accuracy across all folds

print("Accuracy: {:.4f}".format(svm_linear_accuracy))

draw_svm_plot(svm_linear, X_important, y, "Linear Kernel")

Accuracy: 0.9661

C:\Users\pufan\AppData\Roaming\Python\Python311\site-packages\sklearn\base.py:439: UserWarning: X does not have valid feature names, but SVC was fitted with feature names
  warnings.warn(

Polynomial Kernel

The polynomial kernel is a more complex kernel. The formula for the polynomial kernel is: $$ K(x, x') = (x^T x' + 1)^d $$

Let's try to use the polynomial kernel with degree 3.

It uses the feature matrix X_important and target vector y. The code then performs 5-fold cross-validation to evaluate the model's accuracy. The average accuracy across all folds is then printed to the console.

Finally, the function draw_svm_plot() is called to plot the decision boundary of the trained SVM model along with the data points in X_important, colored by their corresponding target values in y. The plot is titled "Polynomial Kernel (Degree 3)".

svm_poly = SVC(random_state=87, kernel="poly", degree=3).fit(X_important, y.squeeze())

scores = cross_val_score(svm_poly, X_important, y.squeeze(), cv=5)
svm_poly_accuracy = scores.mean()

print("Accuracy: {:.4f}".format(svm_poly_accuracy))

draw_svm_plot(svm_poly, X_important, y, "Polynomial Kernel (Degree 3)")

Accuracy: 0.9458

C:\Users\pufan\AppData\Roaming\Python\Python311\site-packages\sklearn\base.py:439: UserWarning: X does not have valid feature names, but SVC was fitted with feature names
  warnings.warn(

Gaussian Kernel

The Gaussian kernel is a more complex kernel. The formula for the Gaussian kernel is: $$ K(x, x') = \exp(-\gamma ||x - x'||^2) $$

This code trains a support vector machine (SVM) model with a Gaussian (RBF) kernel, using the feature matrix X_important and target vector y. The code then performs 5-fold cross-validation to evaluate the model's accuracy. The average accuracy across all folds is then printed to the console.

Finally, the function draw_svm_plot() is called to plot the decision boundary of the trained SVM model along with the data points in X_important, colored by their corresponding target values in y. The plot is titled "Gaussian Kernel".

svm_gaussian = SVC(random_state=87, kernel="rbf").fit(X_important, y.squeeze())

scores = cross_val_score(svm_gaussian, X_important, y.squeeze(), cv=5)
svm_gaussian_accuracy = scores.mean()

print("Accuracy: {:.4f}".format(svm_gaussian_accuracy))

draw_svm_plot(svm_gaussian, X_important, y, "Gaussian Kernel")

Accuracy: 0.9684

C:\Users\pufan\AppData\Roaming\Python\Python311\site-packages\sklearn\base.py:439: UserWarning: X does not have valid feature names, but SVC was fitted with feature names
  warnings.warn(

Sigmoid Kernel

The sigmoid kernel is a more complex kernel. The formula for the sigmoid kernel is: $$ K(x, x') = \tanh(\gamma x^T x' + r) $$

This code trains a support vector machine (SVM) model with a sigmoid kernel, using the feature matrix X_important and target vector y. The code then performs 5-fold cross-validation to evaluate the model's accuracy. The average accuracy across all folds is then printed to the console.

Finally, the function draw_svm_plot() is called to plot the decision boundary of the trained SVM model along with the data points in X_important, colored by their corresponding target values in y. The plot is titled "Sigmoid Kernel".

svm_sigmoid = SVC(random_state=87, kernel="sigmoid").fit(X_important, y.squeeze())

scores = cross_val_score(svm_sigmoid, X_important, y.squeeze(), cv=5)
svm_sigmoid_accuracy = scores.mean()

print("Accuracy: {:.4f}".format(svm_sigmoid_accuracy))

draw_svm_plot(svm_sigmoid, X_important, y, "Sigmoid Kernel")

Accuracy: 0.8654

C:\Users\pufan\AppData\Roaming\Python\Python311\site-packages\sklearn\base.py:439: UserWarning: X does not have valid feature names, but SVC was fitted with feature names
  warnings.warn(

Here are the comparison of the results of the different kernels:

results = pd.DataFrame(
    {
        "Kernel": ["Linear", "Polynomial", "Gaussian", "Sigmoid"],
        "Accuracy": [svm_linear_accuracy, svm_poly_accuracy, svm_gaussian_accuracy, svm_sigmoid_accuracy],
    }
)
results

	Kernel	Accuracy
0	Linear	0.966099
1	Polynomial	0.945824
2	Gaussian	0.968427
3	Sigmoid	0.865405

# Create the bar plot with accuracy values as annotations
ax = sns.barplot(data=results, x="Kernel", y="Accuracy")
ax.set_xlabel("Kernel")
ax.set_ylabel("Accuracy")
ax.set_title("Accuracy of SVM with Different Kernels")

# Add accuracy values as annotations to the bar plot
for i in range(len(results)):
    ax.annotate(f"{results['Accuracy'][i]:.4f}", (i, results['Accuracy'][i]), ha='center', va='bottom')

# Show the plot
plt.show()

Put all the graphs together

def draw_svm_plot_subplot(svm, X, y, title, axes):
    X = np.array(X)
    y = np.array(y == "female")

    axes.set_title(title)
    axes.scatter(X[:, 0], X[:, 1], c=y, s=30, cmap=plt.cm.Paired)
    axes.axis("tight")

    x_min = X[:, 0].min()
    x_max = X[:, 0].max()
    y_min = X[:, 1].min()
    y_max = X[:, 1].max()

    XX, YY = np.mgrid[x_min:x_max:200j, y_min:y_max:200j]
    Z = svm.decision_function(np.c_[XX.ravel(), YY.ravel()]).reshape(XX.shape)

    axes.contour(XX, YY, Z, colors=["k", "k", "k"], linestyles=["--", "-", "--"], levels=[-1, 0, 1], linewidths=[1, 5, 1])
    axes.scatter(svm.support_vectors_[:, 0], svm.support_vectors_[:, 1], s=100, linewidth=0.5, facecolors="none",
                 edgecolors="k")


figure, axes = plt.subplots(2, 2, figsize=(80, 80))
draw_svm_plot_subplot(SVC(random_state=87, kernel="linear").fit(X_important, y.squeeze()), X_important, y.squeeze(), "Linear Kernel", axes[0, 0])
draw_svm_plot_subplot(SVC(random_state=87, kernel="poly", degree=3).fit(X_important, y.squeeze()), X_important, y.squeeze(), "Polynomial Kernel", axes[0, 1])
draw_svm_plot_subplot(SVC(random_state=87, kernel="rbf", gamma=0.1).fit(X_important, y.squeeze()), X_important, y.squeeze(), "Gaussian Kernel", axes[1, 0])
draw_svm_plot_subplot(SVC(random_state=87, kernel="sigmoid", gamma=0.1, coef0=0.1).fit(X_important, y.squeeze()), X_important, y.squeeze(), "Sigmoid Kernel", axes[1, 1])

# make the font size of the title larger
for ax in axes.flat:
    ax.title.set_fontsize(100)

plt.show()

C:\Users\pufan\AppData\Roaming\Python\Python311\site-packages\sklearn\base.py:439: UserWarning: X does not have valid feature names, but SVC was fitted with feature names
  warnings.warn(
C:\Users\pufan\AppData\Roaming\Python\Python311\site-packages\sklearn\base.py:439: UserWarning: X does not have valid feature names, but SVC was fitted with feature names
  warnings.warn(
C:\Users\pufan\AppData\Roaming\Python\Python311\site-packages\sklearn\base.py:439: UserWarning: X does not have valid feature names, but SVC was fitted with feature names
  warnings.warn(
C:\Users\pufan\AppData\Roaming\Python\Python311\site-packages\sklearn\base.py:439: UserWarning: X does not have valid feature names, but SVC was fitted with feature names
  warnings.warn(

After carefully examining the results, we can see that the Gaussian kernel performs the best.

Exploring the Data with All Features

Average accuracy of the linear SVM model with all features

svm_linear = SVC(random_state=87, kernel="linear")
svm_linear_accuracy = cross_val_score(svm_linear, X, y.squeeze(), cv=5).mean()
print("Linear Accuracy: {:.4f}".format(svm_linear_accuracy))

Linear Accuracy: 0.9681

Average accuracy of the Gaussian SVM model with all features

svm_gaussian = SVC(random_state=87, kernel="rbf")
svm_gaussian_accuracy = cross_val_score(svm_gaussian, X, y.squeeze(), cv=5).mean()
print("Gaussian Accuracy: {:.4f}".format(svm_gaussian_accuracy))

Gaussian Accuracy: 0.9694

Average accuracy of the Polynomial SVM model with all features

svm_poly = SVC(random_state=87, kernel="poly", degree=3)
svm_poly_accuracy = cross_val_score(svm_poly, X, y.squeeze(), cv=5).mean()
print("Polynomial Accuracy: {:.4f}".format(svm_poly_accuracy))

Polynomial Accuracy: 0.9568

Average accuracy of the Sigmoid SVM model with all features

svm_sigmoid = SVC(random_state=87, kernel="sigmoid")
svm_sigmoid_accuracy = cross_val_score(svm_sigmoid, X, y.squeeze(), cv=5).mean()
print("Sigmoid Accuracy: {:.4f}".format(svm_sigmoid_accuracy))

Sigmoid Accuracy: 0.8069

SVM with $\gamma$ and $C$ Tuning

We try to use the Gaussian kernel with $\gamma$ and $C$ tuning.

Understanding $\gamma$ and $C$

$\gamma$ and $C$ are parameters of the SVM algorithm. They are used to control the trade-off between the smooth decision boundary and the correct classification of training examples. The $\gamma$ parameter defines how far the influence of a single training example reaches, with low values meaning 'far' and high values meaning 'close'. The $C$ parameter trades off correct classification of training examples against maximization of the decision function's margin. A low $C$ makes the decision surface smooth, while a high $C$ aims at classifying all training examples correctly by giving the model freedom to select more samples as support vectors.

figure, axes = plt.subplots(2, 2, figsize=(80, 80))

draw_svm_plot_subplot(SVC(random_state=87, C=1, gamma=0.01).fit(X_important, y.squeeze()), X_important, y.squeeze(), "C = 1, gamma = 0.01", axes[0, 0])
draw_svm_plot_subplot(SVC(random_state=87, C=100, gamma=0.01).fit(X_important, y.squeeze()), X_important, y.squeeze(), "C = 100, gamma = 0.01", axes[0, 1])
draw_svm_plot_subplot(SVC(random_state=87, C=1, gamma=1).fit(X_important, y.squeeze()), X_important, y.squeeze(), "C = 1, gamma = 0.1", axes[1, 0])
draw_svm_plot_subplot(SVC(random_state=87, C=100, gamma=1).fit(X_important, y.squeeze()), X_important, y.squeeze(), "C = 100, gamma = 0.1", axes[1, 1])


# make the font size of the title larger
for ax in axes.flat:
    ax.title.set_fontsize(100)

plt.show()

C:\Users\pufan\AppData\Roaming\Python\Python311\site-packages\sklearn\base.py:439: UserWarning: X does not have valid feature names, but SVC was fitted with feature names
  warnings.warn(
C:\Users\pufan\AppData\Roaming\Python\Python311\site-packages\sklearn\base.py:439: UserWarning: X does not have valid feature names, but SVC was fitted with feature names
  warnings.warn(
C:\Users\pufan\AppData\Roaming\Python\Python311\site-packages\sklearn\base.py:439: UserWarning: X does not have valid feature names, but SVC was fitted with feature names
  warnings.warn(
C:\Users\pufan\AppData\Roaming\Python\Python311\site-packages\sklearn\base.py:439: UserWarning: X does not have valid feature names, but SVC was fitted with feature names
  warnings.warn(

This code performs a grid search to find the best hyperparameters for a support vector machine (SVM) model with a Gaussian (RBF) kernel. The hyperparameters being optimized are the regularization parameter $C$ and the kernel coefficient gamma.

The code defines a parameter grid containing a range of values for both hyperparameters to search over. A new SVM model is created with a Gaussian kernel, and the GridSearchCV class is used to perform a 5-fold cross-validated grid search over the parameter grid.

The best hyperparameters and corresponding accuracy are printed to the console, which shows the best combination of $C$ and $\gamma$ values that maximizes the accuracy of the SVM model on the training set.

from sklearn.svm import SVC
from sklearn.model_selection import GridSearchCV
import numpy as np

# define parameter ranges to search over
param_grid = {'C': [0.1, 1, 10, 100],
              'gamma': [0.01, 0.1, 1, 10]}

# create SVM model with Gaussian kernel
svm = SVC(kernel='rbf')

# use grid search to find best hyperparameters
grid_search = GridSearchCV(svm, param_grid, cv=5)
grid_search.fit(X, y.squeeze())

# print best hyperparameters and corresponding accuracy
print("Best hyperparameters: ", grid_search.best_params_)
print("Best accuracy: {:.4f}".format(grid_search.best_score_))

Best hyperparameters:  {'C': 10, 'gamma': 0.01}
Best accuracy: 0.9701

Then, we use hill climbing to find the best hyperparameters for the SVM model.

import numpy as np
from sklearn.svm import SVC
from sklearn.model_selection import cross_val_score

# Define the objective function
def objective(X, y, gamma, C):
    clf = SVC(kernel='rbf', gamma=gamma, C=C)
    return cross_val_score(clf, X, y, cv=5).mean()

# Define the hill climbing algorithm
def hill_climbing(X, y, gamma, C, gamma_step=0.0001, C_step=0.001):
    best_score = objective(X, y, gamma, C)
    best_gamma, best_C = gamma, C
    
    for _ in range(10):
        # Perturb gamma and C
        new_gamma = gamma + np.random.uniform(low=-gamma_step, high=gamma_step)
        new_gamma = max(new_gamma, 0.001)
        new_C = C + np.exp(np.random.uniform(low=-C_step, high=C_step))
        
        # Evaluate the new solution
        new_score = objective(X, y, new_gamma, new_C)
        
        # Update the best solution
        if new_score > best_score:
            best_score = new_score
            best_gamma, best_C = new_gamma, new_C

    return best_gamma, best_C, best_score

We use the same parameter grid as before, but this time we use the RandomizedSearchCV class to perform a 5-fold cross-validated randomized search over the parameter grid. The best hyperparameters and corresponding accuracy are printed to the console, which shows the best combination of $C$ and $\gamma$ values that maximizes the accuracy of the SVM model on the training set.

# Initialize gamma and C using cross-validation
scores = []
for gamma in [0.01, 0.1, 0.2, 0.3, 0.4, 0.5, 0.6, 0.7, 0.8, 0.9, 1]:
    for C in [0.1, 1, 10, 15, 20]:
        score = cross_val_score(SVC(kernel='rbf', gamma=gamma, C=C), X, y.squeeze(), cv=5).mean()
        scores.append((score, gamma, C))
best_score, gamma, C = max(scores)

print(scores)
print(f"Best score: {best_score:.4f} (gamma={gamma:.2f}, C={C:.2f})")

[(0.9461650294912687, 0.01, 0.1), (0.9687597083487655, 0.01, 1), (0.9700891647917921, 0.01, 10), (0.9694236073874659, 0.01, 15), (0.9684258240695186, 0.01, 20), (0.9564568465624845, 0.1, 0.1), (0.965764147240756, 0.1, 1), (0.9581179761305908, 0.1, 10), (0.9544607271380479, 0.1, 15), (0.9541279484358848, 0.1, 20), (0.9385160944383945, 0.2, 0.1), (0.9611096677188076, 0.2, 1), (0.9534629438201006, 0.2, 10), (0.9537951697337217, 0.2, 15), (0.9537951697337217, 0.2, 20), (0.9149214211087833, 0.3, 0.1), (0.9478156560770807, 0.3, 1), (0.9434934024687536, 0.3, 10), (0.9434934024687536, 0.3, 15), (0.9434934024687536, 0.3, 20), (0.8760250081536309, 0.4, 0.1), (0.9345188804926451, 0.4, 1), (0.9341855490019404, 0.4, 10), (0.9341855490019404, 0.4, 15), (0.9341855490019404, 0.4, 20), (0.8128749979270429, 0.5, 0.1), (0.9202198992819277, 0.5, 1), (0.9208843511091702, 0.5, 10), (0.9208843511091702, 0.5, 15), (0.9208843511091702, 0.5, 20), (0.7567216322740062, 0.6, 0.1), (0.9079175902841886, 0.6, 1), (0.9069192541776994, 0.6, 10), (0.9069192541776994, 0.6, 15), (0.9069192541776994, 0.6, 20), (0.7141945041763174, 0.7, 0.1), (0.8879691101762841, 0.7, 1), (0.8889663407056899, 0.7, 10), (0.8889663407056899, 0.7, 15), (0.8889663407056899, 0.7, 20), (0.68994035411634, 0.8, 0.1), (0.8756695651212543, 0.8, 1), (0.8766656900735761, 0.8, 10), (0.8766656900735761, 0.8, 15), (0.8766656900735761, 0.8, 20), (0.6793174167085866, 0.9, 0.1), (0.8670289274243924, 0.9, 1), (0.8676928264630931, 0.9, 10), (0.8676928264630931, 0.9, 15), (0.8676928264630931, 0.9, 20), (0.7079546271164892, 1, 0.1), (0.8547343574662385, 1, 1), (0.8570599388615874, 1, 10), (0.8570599388615874, 1, 15), (0.8570599388615874, 1, 20)]
Best score: 0.9701 (gamma=0.01, C=10.00)

We do the hill climbing for 100 times and get the best hyperparameters.

# Perform hill climbing
for i in range(100):
    gamma, C, score = hill_climbing(X, y.squeeze(), gamma, C, gamma_step=0.01, C_step=10)
    print(f"Iteration {i+1}: gamma={gamma:.4f}, C={C:.2f}, score={score:.4f}")

Iteration 1: gamma=0.0026, C=23.88, score=0.9704
Iteration 2: gamma=0.0026, C=23.88, score=0.9704
Iteration 3: gamma=0.0026, C=23.88, score=0.9704
Iteration 4: gamma=0.0026, C=23.88, score=0.9704
Iteration 5: gamma=0.0026, C=23.88, score=0.9704
Iteration 6: gamma=0.0026, C=23.88, score=0.9704
Iteration 7: gamma=0.0026, C=23.88, score=0.9704
Iteration 8: gamma=0.0026, C=23.88, score=0.9704
Iteration 9: gamma=0.0026, C=23.88, score=0.9704
Iteration 10: gamma=0.0026, C=23.88, score=0.9704
Iteration 11: gamma=0.0026, C=23.88, score=0.9704
Iteration 12: gamma=0.0026, C=23.88, score=0.9704
Iteration 13: gamma=0.0026, C=23.88, score=0.9704
Iteration 14: gamma=0.0026, C=23.88, score=0.9704
Iteration 15: gamma=0.0026, C=23.88, score=0.9704
Iteration 16: gamma=0.0026, C=23.88, score=0.9704
Iteration 17: gamma=0.0026, C=23.88, score=0.9704
Iteration 18: gamma=0.0026, C=23.88, score=0.9704
Iteration 19: gamma=0.0026, C=23.88, score=0.9704
Iteration 20: gamma=0.0026, C=23.88, score=0.9704
Iteration 21: gamma=0.0026, C=23.88, score=0.9704
Iteration 22: gamma=0.0026, C=23.88, score=0.9704
Iteration 23: gamma=0.0026, C=23.88, score=0.9704
Iteration 24: gamma=0.0026, C=23.88, score=0.9704
Iteration 25: gamma=0.0026, C=23.88, score=0.9704
Iteration 26: gamma=0.0026, C=23.88, score=0.9704
Iteration 27: gamma=0.0026, C=23.88, score=0.9704
Iteration 28: gamma=0.0026, C=23.88, score=0.9704
Iteration 29: gamma=0.0026, C=23.88, score=0.9704
Iteration 30: gamma=0.0026, C=23.88, score=0.9704
Iteration 31: gamma=0.0026, C=23.88, score=0.9704
Iteration 32: gamma=0.0026, C=23.88, score=0.9704
Iteration 33: gamma=0.0026, C=23.88, score=0.9704
Iteration 34: gamma=0.0026, C=23.88, score=0.9704
Iteration 35: gamma=0.0026, C=23.88, score=0.9704
Iteration 36: gamma=0.0026, C=23.88, score=0.9704
Iteration 37: gamma=0.0026, C=23.88, score=0.9704
Iteration 38: gamma=0.0026, C=23.88, score=0.9704
Iteration 39: gamma=0.0026, C=23.88, score=0.9704
Iteration 40: gamma=0.0026, C=23.88, score=0.9704
Iteration 41: gamma=0.0026, C=23.88, score=0.9704
Iteration 42: gamma=0.0026, C=23.88, score=0.9704
Iteration 43: gamma=0.0026, C=23.88, score=0.9704
Iteration 44: gamma=0.0026, C=23.88, score=0.9704
Iteration 45: gamma=0.0026, C=23.88, score=0.9704
Iteration 46: gamma=0.0026, C=23.88, score=0.9704
Iteration 47: gamma=0.0026, C=23.88, score=0.9704
Iteration 48: gamma=0.0026, C=23.88, score=0.9704
Iteration 49: gamma=0.0026, C=23.88, score=0.9704
Iteration 50: gamma=0.0026, C=23.88, score=0.9704
Iteration 51: gamma=0.0026, C=23.88, score=0.9704
Iteration 52: gamma=0.0026, C=23.88, score=0.9704
Iteration 53: gamma=0.0026, C=23.88, score=0.9704
Iteration 54: gamma=0.0026, C=23.88, score=0.9704
Iteration 55: gamma=0.0026, C=23.88, score=0.9704
Iteration 56: gamma=0.0026, C=23.88, score=0.9704
Iteration 57: gamma=0.0026, C=23.88, score=0.9704
Iteration 58: gamma=0.0026, C=23.88, score=0.9704
Iteration 59: gamma=0.0026, C=23.88, score=0.9704
Iteration 60: gamma=0.0026, C=23.88, score=0.9704
Iteration 61: gamma=0.0026, C=23.88, score=0.9704
Iteration 62: gamma=0.0026, C=23.88, score=0.9704
Iteration 63: gamma=0.0026, C=23.88, score=0.9704
Iteration 64: gamma=0.0026, C=23.88, score=0.9704
Iteration 65: gamma=0.0026, C=23.88, score=0.9704
Iteration 66: gamma=0.0026, C=23.88, score=0.9704
Iteration 67: gamma=0.0026, C=23.88, score=0.9704
Iteration 68: gamma=0.0026, C=23.88, score=0.9704
Iteration 69: gamma=0.0026, C=23.88, score=0.9704
Iteration 70: gamma=0.0026, C=23.88, score=0.9704
Iteration 71: gamma=0.0026, C=23.88, score=0.9704
Iteration 72: gamma=0.0026, C=23.88, score=0.9704
Iteration 73: gamma=0.0026, C=23.88, score=0.9704
Iteration 74: gamma=0.0026, C=23.88, score=0.9704
Iteration 75: gamma=0.0026, C=23.88, score=0.9704
Iteration 76: gamma=0.0026, C=23.88, score=0.9704
Iteration 77: gamma=0.0026, C=23.88, score=0.9704
Iteration 78: gamma=0.0026, C=23.88, score=0.9704
Iteration 79: gamma=0.0026, C=23.88, score=0.9704
Iteration 80: gamma=0.0026, C=23.88, score=0.9704
Iteration 81: gamma=0.0026, C=23.88, score=0.9704
Iteration 82: gamma=0.0026, C=23.88, score=0.9704
Iteration 83: gamma=0.0026, C=23.88, score=0.9704
Iteration 84: gamma=0.0026, C=23.88, score=0.9704
Iteration 85: gamma=0.0026, C=23.88, score=0.9704
Iteration 86: gamma=0.0026, C=23.88, score=0.9704
Iteration 87: gamma=0.0026, C=23.88, score=0.9704
Iteration 88: gamma=0.0026, C=23.88, score=0.9704
Iteration 89: gamma=0.0026, C=23.88, score=0.9704
Iteration 90: gamma=0.0026, C=23.88, score=0.9704
Iteration 91: gamma=0.0026, C=23.88, score=0.9704
Iteration 92: gamma=0.0026, C=23.88, score=0.9704
Iteration 93: gamma=0.0026, C=23.88, score=0.9704
Iteration 94: gamma=0.0026, C=23.88, score=0.9704
Iteration 95: gamma=0.0026, C=23.88, score=0.9704
Iteration 96: gamma=0.0026, C=23.88, score=0.9704
Iteration 97: gamma=0.0026, C=23.88, score=0.9704
Iteration 98: gamma=0.0026, C=23.88, score=0.9704
Iteration 99: gamma=0.0026, C=23.88, score=0.9704
Iteration 100: gamma=0.0026, C=23.88, score=0.9704

And here is the final result:

# Train the final SVM model
clf = SVC(kernel='rbf', gamma=gamma, C=C)
print("gamma: {:.6f}, C: {:.6f}".format(gamma, C))
print("Final accuracy: {:.10f}".format(cross_val_score(clf, X, y.squeeze(), cv=5).mean()))

gamma: 0.002601, C: 23.877762
Final accuracy: 0.9704219435